Welcome to Apache Kafka Overview. After watching this video, you will be able to: Recognize Apache Kafka as an
Event Streaming Platform (ESP). Describe the architecture of Apache Kafka. List common use cases for Apache Kafka. Summarize the main features and
benefits of Apache Kafka, and list popular Kafka-based
ESP-as-a-Service providers. By now you should be able to describe what an
Event Streaming Platform and its core components are, and may think that it must be extremely
difficult to implement an ESP from scratch. That is true, but we already
have many great open source and commercial ESP solutions available
that come with many built-in capabilities. Among these, Apache Kafka, an open source project, has probably become THE most
popular event streaming platform. Kafka is a comprehensive platform and can
be used in many application scenarios. Let's look at some common ones here: Kafka was originally used to track user
activities such as keyboard strokes, mouse clicks, page views, searches,
gestures, screen time, and so on. But now it is also suitable for all kinds
of metric-streaming such as sensor readings, GPS, as well as hardware and software monitoring. For enterprise applications and
infrastructure with a huge number of logs, Kafka can be used to collect and integrate
them into a centralized repository. For banks, insurance, or FinTech companies, Kafka
is also widely used for payments and transactions. These scenarios are just the tip of the iceberg. You basically can use Kafka in scenarios
when you want high throughput and reliable data transportation services among
various event sources and destinations. All these events will be ingested
in Kafka, and become available for subscriptions and consumption including: Further data storage and movement to other
online or offline databases and backups. Real-time processing and analytics, including dashboard, machine
learning, AI algorithms, and so on. Generating notifications such as email,
text messages, and instant messages, or data governance and auditing to
make sure sensitive data such as bank transactions are complying with regulations. Kafka has a distributed
client-server architecture: For the server side, Kafka is a
cluster with many associated servers called brokers, acting as
the event broker to receive, store, and distribute events. All those brokers
are managed by another distributed system called ZooKeeper to ensure all brokers work
in an efficient and collaborative way. Kafka uses a TCP (Transmission
Control Protocol) based network communication protocol to exchange
data between clients and servers. For the client side, Kafka provides
different types of clients, such as Kafka CLI (Command Line Interface) which is a collection of shell scripts to
communicate with the Kafka server, many high-level programming
APIs such as Java, Scala, REST APIs, and some specific third party clients
made by the Kafka community. Accordingly, you could choose different
clients based on your requirements. Now that you have a basic understanding of Kafka, let’s see why Apache Kafka
has become such a popular ESP: Kafka has the following main features: It is a distribution system which makes it highly scalable to handle high data
throughput and concurrency. A Kafka cluster normally has multiple event
brokers which can handle event streaming in parallel. As such, Kafka is
very fast and highly scalable. Kafka also divides event storage into
multiple partitions and replications which makes it fault-tolerant and highly reliable. Kafka stores the events permanently. As such, event consumption could be done whenever
suitable for consumers without a deadline. And, Kafka is open source, meaning
that you can use it for free, and even customize it based
on your specific requirements. Even though Kafka is open source
and well-documented, it is still challenging to configure and deploy a
Kafka without professional assistance. Deploying a Kafka cluster requires
extensive efforts for tuning infrastructure and consistently adjusting the configurations,
especially for enterprise-level deployments. Fortunately, several commercial service providers offer an on-demand ESP-as-a-Service
to meet your streaming requirements. Many of them are built on top of Kafka
and provide added value for customers. Some well-known ESP providers include: Confluent Cloud, which provides customers
with fully managed Kafka services, either on-premises or on-cloud. IBM Event Streams, which is also based on
Kafka and provides many add-on services such as enterprise-grade security, disaster
recovery, and 24-7 cluster monitoring. Amazon Managed Streaming for Apache Kafka, which is also a fully managed service to
facilitate the build and deployment of Kafka. In this video, you learned that: Apache Kafka is one of the
most popular open source ESPs. Common Kafka use cases include
user-activity tracking, metrics and logs integrations, and
financial transaction processing. Apache Kafka is a highly scalable and reliable
platform that stores events permanently. And finally, popular Kafka-based ESP
service providers include Confluent Cloud, IBM Event Streams, and Amazon Managed Streaming.